feat: integrate metrics into live sync process by meyer9 · Pull Request #330 · op-rs/op-reth

meyer9 · 2025-10-28T14:48:32Z

Fixes #246

Return UpdateResult from store_trie_updates with counts of items stored.
Log counts and latencies for live collector

Copilot

Pull Request Overview

This PR integrates metrics collection into the live sync process by introducing an UpdateResult struct that tracks counts of items stored during trie updates. The changes enable better observability of the sync process by logging counts and latencies.

Key Changes:

Introduced UpdateResult struct to return counts of account trie updates, storage trie updates, hashed accounts, and hashed storages written
Updated store_trie_updates method signature across all implementations to return UpdateResult
Added metrics recording in LiveTrieCollector to track execution duration, state root calculation, write duration, and update counts

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
crates/optimism/trie/src/api.rs	Defines new `UpdateResult` struct and updates `OpProofsStore` trait signature
crates/optimism/trie/src/db/store.rs	Implements `UpdateResult` return type in MDBX storage implementation
crates/optimism/trie/src/in_memory.rs	Implements `UpdateResult` return type in in-memory storage with counting logic
crates/optimism/trie/src/live.rs	Integrates metrics recording for block processing durations and update counts
crates/optimism/trie/src/metrics.rs	Removes unused block-level metrics field and updates import for `UpdateResult`

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-28T15:53:15Z

crates/optimism/trie/src/api.rs

+#[derive(Debug, Clone, Default)]
+/// Counts from storing trie updates and post state.


The doc comment should precede the derive attributes according to Rust conventions. Move the comment above the #[derive(...)] line.

Suggested change

#[derive(Debug, Clone, Default)]

/// Counts from storing trie updates and post state.

/// Counts from storing trie updates and post state.

#[derive(Debug, Clone, Default)]

emhane · 2025-10-29T11:59:13Z

crates/optimism/trie/src/in_memory.rs

+        let mut num_account_trie_updates_written = 0;
+        let mut num_storage_trie_updates_written = 0;
+        let mut num_hashed_accounts_written = 0;
+        let mut num_hashed_storages_written = 0;
+
        // Store account branch nodes
        for (path, branch) in block_state_diff.trie_updates.account_nodes_ref() {
            self.account_branches.insert((block_number, *path), Some(branch.clone()));
+            num_account_trie_updates_written += 1;


how about

Suggested change

let mut num_account_trie_updates_written = 0;

let mut num_storage_trie_updates_written = 0;

let mut num_hashed_accounts_written = 0;

let mut num_hashed_storages_written = 0;

// Store account branch nodes

for (path, branch) in block_state_diff.trie_updates.account_nodes_ref() {

self.account_branches.insert((block_number, *path), Some(branch.clone()));

num_account_trie_updates_written += 1;

let mut acc = UpdateResult::default();

// Store account branch nodes

for (path, branch) in block_state_diff.trie_updates.account_nodes_ref() {

self.account_branches.insert((block_number, *path), Some(branch.clone()));

acc.num_account_trie_updates_written += 1;

nice, this is cleaner!

emhane · 2025-10-29T12:03:06Z

crates/optimism/trie/src/live.rs

+    #[cfg(feature = "metrics")]
+    metrics: BlockMetrics,
    provider: Provider,
    storage: &'tx OpProofsStorage<PreimageStore>,


this addition is redundant since the block metrics are already stored in OpProofsStorage accessible via self.storage.metrics().block_metrics() when the metrics feature is enabled

ah I see, fixed now!

emhane · 2025-10-29T12:09:21Z

crates/optimism/trie/src/live.rs

+        #[cfg(feature = "metrics")]
+        {
+            self.metrics.execution_duration_seconds.record(execute_block_duration);
+            self.metrics.state_root_duration_seconds.record(calculate_state_root_duration);
+            self.metrics.write_duration_seconds.record(write_trie_updates_duration);
+            self.metrics.total_duration_seconds.record(execute_and_store_total_duration);
+            self.metrics
+                .account_trie_updates_written_total
+                .increment(update_result.account_trie_updates_written);
+            self.metrics
+                .storage_trie_updates_written_total
+                .increment(update_result.storage_trie_updates_written);
+            self.metrics
+                .hashed_accounts_written_total
+                .increment(update_result.hashed_accounts_written);
+            self.metrics
+                .hashed_storages_written_total
+                .increment(update_result.hashed_storages_written);
+        }


let's make intermediary types in metrics module

pub struct OperationDurations { pub execute_block: Duration, pub calculate_state_root: Duration, ... }

and

pub struct WriteCounts { pub account_trie_updates: usize, ... }

and then add two methods OpProofsStorageWithMetrics::record_operation_durations(&self, durations: OperationDurations) and OpProofsStorageWithMetrics::increment_write_counts(&self, counts: WriteCounts)

updated! much cleaner

emhane

lgtm

Signed-off-by: Julian Meyer <julian.meyer@coinbase.com>

Fixes #246 - Return `UpdateResult` from `store_trie_updates` with counts of items stored. - Log counts and latencies for live collector Signed-off-by: Julian Meyer <julian.meyer@coinbase.com>

Fixes op-rs/op-reth#246 - Return `UpdateResult` from `store_trie_updates` with counts of items stored. - Log counts and latencies for live collector Signed-off-by: Julian Meyer <julian.meyer@coinbase.com>

@dhyaniarun1993

…19194) * refactor(db): derive Clone for DatabaseEnv (paradigmxyz/reth#21641) Co-authored-by: Amp <amp@ampcode.com> * chore(deps): breaking bumps (paradigmxyz/reth#21584) Co-authored-by: Georgios Konstantopoulos <me@gakonst.com> Co-authored-by: Amp <amp@ampcode.com> * refactor: add with_* compressor utility methods (paradigmxyz/reth#21680) * refactor: use alloy_primitives::map for all HashMap/HashSet types (paradigmxyz/reth#21686) Co-authored-by: Amp <amp@ampcode.com> * fix: support EIP-1559 params configuration for Optimism dev mode (paradigmxyz/reth#21855) * chore(rust/op-reth): pull latest changes and update dependencies * chore(op-exex): Add crate `reth-optimism-exex` and `reth-optimism-trie` (op-rs/op-reth#204) - Adds new crates `reth-optimism-exex` and `reth_optimism_trie` - Moves `reth_exex::external_proofs` -> `reth_optimism_exex` - Moves `reth_exex::external_proofs::storage` -> `reth_optimism_trie` * fix: ci lint and doc (op-rs/op-reth#234) Seems like when `reth-optimism-trie` depends on `reth-db-api` indirectly, the doctest crate couldn’t resolve `reth_db_api::…` and failing. Closes op-rs/op-reth#233 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: separate storage and account cursors (op-rs/op-reth#229) This PR separates storage and account cursors which makes sense because these two cursors may have very different seek/next implementations. In the case of in-memory, a single cursor impl can still handle both, but for MDBX, it makes sense to implement separate cursors. Closes op-rs/op-reth#236 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: add external state provider implementation (op-rs/op-reth#197) Based on op-rs/op-reth#203 , op-rs/op-reth#204 This PR implements `StateProvider` given a `OpProofsStorage` instance. It reads most data from the external database, but falls back on the latest provider for block hashes and code by hash similar to the existing historical provider in Reth. This is an important part to implementing live syncing since we're running the sync process on the DB being created. In the Reth implementation, `proof.rs` is contained in `trie/db`, so I think it makes sense to go in our DB crate. The `provider.rs` is in the `reth-provider` crate, but I don't think a separate provider crate helps us here, so I think we should also include that in the trie crate as well. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: add deletion tests for op proofs storage (op-rs/op-reth#230) Based on op-rs/op-reth#229 Adds tests to ensure that storing TrieUpdates that include deletions actually deletes the nodes at that block height, and that updates take precedence over deletions (same as `write_trie_updates` in Reth). Closes op-rs/op-reth#238 * feat: implement live state collector using external provider (op-rs/op-reth#198) Based on op-rs/op-reth#197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: add binary entry point for external proofs in OP (op-rs/op-reth#222) Closes op-rs/op-reth#164 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: store hashed account and storage (op-rs/op-reth#235) Closes op-rs/op-reth#211 * feat: implement cursor read/write metrics (op-rs/op-reth#232) Fixes op-rs/op-reth#225 (does not track errors as these are fatal currently) * feat: store account and storage branches (op-rs/op-reth#255) Closes op-rs/op-reth#210 * feat: Concrete error variants for `reth-optimism-trie` (op-rs/op-reth#251) Fixes op-rs/op-reth#237 Adds more concrete error variants for `OpProofsStorageError` Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: Remove block_number from initial sync store methods (op-rs/op-reth#248) Fixes op-rs/op-reth#247 Removes the `block_number` parameter from: - store_account_branches - store_storage_branches - store_hashed_accounts - store_hashed_storages Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat(MDBX): Implement earliest and latest block tracking in storage (op-rs/op-reth#253) Fixes op-rs/op-reth#215 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat(deps): Add `serde-bincode-compat` feature to `reth-trie` (op-rs/op-reth#250) Ref op-rs/op-reth#241 - Fixes comment from deps fix in op-rs/op-reth#229 (fix correct but comment misleading) - `serde-bincode-compat` is a feature which must be enabled when the `bincode` dependency is used. Asides for dev-deps, this only happens in `reth-stages`. `reth-exex` runs in `reth-stages`. In this wokrstream we use the `TrieUpdate` in exex. `serde-bincode-compat` is already implemented for `TrieUpdates` in `reth-trie-common`, this PR simply makes that feature accessible via `reth-trie` and in turn `reth-optimism-trie`, in order to enable it in `reth-optimism-exex`. * feat: store trie updates (op-rs/op-reth#258) Closes: op-rs/op-reth#214 * feat: implemented mdbx trie cursor (op-rs/op-reth#249) Closes op-rs/op-reth#212 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore(deps): Rm redundant dep (op-rs/op-reth#260) Ref op-rs/op-reth#205 Removes redundant dep `reth-db-api` from `reth-optimism-trie`, since used types are re-exported via `reth-db` --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * refactor: replace tuple with `BlockNumHash` (op-rs/op-reth#254) Resolves op-rs/op-reth#239 --------- Co-authored-by: Einar Rasmussen <einar@oplabs.co> * feat: Return ref to B256 (op-rs/op-reth#272) Fixes op-rs/op-reth#270 * chore(exex): Add missing crate header to `reth-optimism-exex` (op-rs/op-reth#271) Resolves op-rs/op-reth#243 --------- Co-authored-by: Einar Rasmussen <einar@oplabs.co> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: implemented hashed cursors (op-rs/op-reth#268) Closes op-rs/op-reth#213 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: add support for `eth_getProof` (op-rs/op-reth#257) Adds support for `eth_getProof` RPC method. This required reworking the launch command to still work with an alternative DB provider. Closes op-rs/op-reth#173 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: implement proof collection e2e tests (op-rs/op-reth#231) Implements end to end tests for proof collection. These tests create a blockchain, run the backfill job, then run the live collector. The live collector executes on top of the backfilled state and it already checks that the state root matches, so the tests just ensure that it runs without errors. Fixes op-rs/op-reth#170 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * perf(trie): feature gate `reth-optimism-trie` metrics (op-rs/op-reth#282) Closes op-rs/op-reth#281 - Feature gates `reth-optimism-trie` metrics - Moves cursor impls out of proofs module into new module `cursor` - Moves cursor factory impls into new module `cursor_factory` - Updates cursor factory impls to return cursor types with metrics wrapper if metrics feature is enabled * chore(docs): Add missing doclinks (op-rs/op-reth#277) Adds doc links to all types to make sure they are caught by lint if names of types change in the future * fix(docs): Fixes docs of `reth_optimism_trie::cursor_factory` (op-rs/op-reth#286) Fixes module docs of `reth_optimism_trie::cursor_factory`, which weren't updated on moving the code out of the proofs module * fix(test): Enable live collector tests with metrics feature (op-rs/op-reth#291) Closes op-rs/op-reth#283 Enable live collector tests when metrics feature is enabled * fix(docs): Fixes doc links (op-rs/op-reth#292) Fixes some doc links from op-rs/op-reth#277 Several places should have linked to `OpProofsStore` which linked to `OpProofsStorage` * feat: Track eth_getProof metrics (op-rs/op-reth#285) Adds latency, total requests, successful responses, and error responses metrics to the eth_getProof RPC method. closes op-rs/op-reth#226 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: devnet setup (op-rs/op-reth#295) Closes op-rs/op-reth#200 * feat: implement `debug_executePayload` (op-rs/op-reth#276) Closes op-rs/op-reth#189 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: storage trie cursor (op-rs/op-reth#301) Closes op-rs/op-reth#299 * feat: latest block updates in `store_trie_updates` (op-rs/op-reth#304) Closes op-rs/op-reth#302 * feat: live collector integration (op-rs/op-reth#306) Closes op-rs/op-reth#296 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: add mdbx to storage tests (op-rs/op-reth#308) Fixes a small issue where storage slots with a zero value weren't skipped. I implemented this above the BlockVersionedCursor since it only applies to the storage cursor. Also enables MDBX in the common storage tests and ensures that no storage tests insert block numbers out of order or insert duplicate keys. * fix: merge conflict caused by storage tests (op-rs/op-reth#313) Fix merge conflicts caused by storage API change. * feat: create index table to efficiently prune history tables (op-rs/op-reth#280) Closes op-rs/op-reth#219 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Julian Meyer <julianmeyer2000@gmail.com> * chore(trie): Move `OpProofsStorageError` to own module (op-rs/op-reth#317) Ref op-rs/op-reth#316 Moves `OpProofsStorageError` into own module for readability Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: Named fields for OpProofsStorageError variants (op-rs/op-reth#319) Fixes op-rs/op-reth#315 --------- Co-authored-by: Emilia Hane <emiliaha95@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: implement `debug_executionWitness` (op-rs/op-reth#312) Add support for `debug_executionWitness` API. This hasn't been tested, but I'll be able to test it soon once the backfill job completes. This code is mostly copied from here and adapted to work with our state provider: https://github.com/op-rs/op-reth/blob/1470a9cc77af798c999da05d9541a7025e5d4924/crates/rpc/rpc/src/debug.rs#L643-L695 Fixes op-rs/op-reth#190 * chore(trie): Rename metrics scope (op-rs/op-reth#322) Ref op-rs/op-reth#321 Renames scope for `BlockMetrics` and `OperationMetrics` from `external_proofs` to `optimism_trie` * feat: implemented wiped storage handling on `store_trie_updates` (op-rs/op-reth#300) Closes op-rs/op-reth#259 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore(trie): Rename `OpProofsStorageError` variant (op-rs/op-reth#324) Ref op-rs/op-reth#315 Renames `OpProofsStorageError::BlockUpdateFailed` to `OpProofsStorageError::MissingParentBlock` which better matches the error message Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * fix(db): `is_parent check` in `store_trie_update` (op-rs/op-reth#309) Fixes op-rs/op-reth#305 * feat: fetch trie updates (op-rs/op-reth#327) Closes: op-rs/op-reth#256 --------- Co-authored-by: itschaindev <jagrutk@protonmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: jagroot <4516586+itschaindev@users.noreply.github.com> * feat: implemented mdbx `replace_updates` (op-rs/op-reth#329) Closes op-rs/op-reth#193 Closes op-rs/op-reth#216 * feat: apply `BlockStateDiff` to the `prune_earliest_state` method (op-rs/op-reth#332) Closes op-rs/op-reth#217 Also fixed a bug where the `set_earliest_block_number_hash` was using `append` instead of `upsert`. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: refactor storage trie cursor (op-rs/op-reth#338) Closes op-rs/op-reth#337 * feat: integrate metrics into live sync process (op-rs/op-reth#330) Fixes op-rs/op-reth#246 - Return `UpdateResult` from `store_trie_updates` with counts of items stored. - Log counts and latencies for live collector Signed-off-by: Julian Meyer <julian.meyer@coinbase.com> * test: basic storage proof (op-rs/op-reth#336) Part 1 for op-rs/op-reth#289 * test: `eth_getProof` for account proof validation (op-rs/op-reth#341) Closes op-rs/op-reth#288 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * test: `debug_executePayload` RPC (op-rs/op-reth#340) Closes op-rs/op-reth#325 * test: multi slot and zero testing (op-rs/op-reth#343) Closes op-rs/op-reth#289 * test: resyncing (op-rs/op-reth#350) Closes op-rs/op-reth#349 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * test: e2e for `debug_executionWitness` (op-rs/op-reth#348) Closes op-rs/op-reth#326 Fixed the `nonce too low` issue with the way `execution_witness` was implemented for op. * test: doc test eg for exex setup (op-rs/op-reth#352) Closes op-rs/op-reth#342 * chore: enable remaining mdbx tests (op-rs/op-reth#354) Fixes op-rs/op-reth#252 * fix: ensure db tx is closed after backfill (op-rs/op-reth#357) The db tx isn't dropped ever for the backfill job, so create a temp scope to ensure it's dropped. * refactor: `MdbxProofStorage` (op-rs/op-reth#351) Closes op-rs/op-reth#335 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * fix: rebase conflicts (op-rs/op-reth#367) Fix conflicts rebasing onto latest upstream main * feat: handle reorg (op-rs/op-reth#356) Closes op-rs/op-reth#191 --------- * feat: add target debug logs to reth_optimism_exex (op-rs/op-reth#368) Fixes op-rs/op-reth#366 * test: basic reorg (op-rs/op-reth#369) Part of op-rs/op-reth#174 * fix: missing block after reorg (op-rs/op-reth#382) Closes op-rs/op-reth#379 * feat: add sync status RPC (op-rs/op-reth#353) Adds `debug_proofsSyncStatus` for fetching the proof window when enabled. Also adds metrics for the earliest and latest block numbers available. * feat: replace variant OpProofsStorageError::Other (op-rs/op-reth#355) Fixes op-rs/op-reth#316 I also noticed a few `DatabaseError` that were being mapped to `OpProofsStorageError::Other` and changed them to point to `OpProofsStorageError::DatabaseError` --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: Implemented `OpProofStoragePruner` (op-rs/op-reth#360) Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: handle `ChainReverted` notification (op-rs/op-reth#380) Closes op-rs/op-reth#192 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: small lint (op-rs/op-reth#390) Find a small typo here. * test: reorg test scenarios (op-rs/op-reth#384) Closes op-rs/op-reth#174 * test: add new devnet for op-reth as sequencer (op-rs/op-reth#391) Closes op-rs/op-reth#389 * chore(trie): Simplify cursor interface (op-rs/op-reth#388) Ref op-rs/op-reth#371 Removes redundant `OpProofsTrieCursorRO` and `OpProofsHashedCursorRO` traits by enabling a `DatabaseError` variant that wraps an error trait object. This way we can have a bijective relation between `DatabaseError` and `OpProofsStorageError` in a type safe way, i.e. conversion both ways without matching on string. cc @dhyaniarun1993 * feat: add panels for `eth_getProof` in grafana dashboard (op-rs/op-reth#392) Closes op-rs/op-reth#318 <img width="2208" height="672" alt="Screenshot 2025-11-21 at 15 35 16" src="https://github.com/user-attachments/assets/62ba3b36-fd4f-498a-9bbc-7e4f33c9e3d2" /> The OP-Reth dashboard is not loaded by default when running the kurtosis devnet for e2e historical proofs. Once op-rs/op-reth#391 is merged, I'll update the devnet config to load it by default. * fix: add custom TrieType to separate metrics for op proofs storage (op-rs/op-reth#397) Closes op-rs/op-reth#321 * feat: implement methods for new traits for `inmemory` cursor (op-rs/op-reth#396) Closes op-rs/op-reth#400 * fix: use loop instead of recursing in hashed next (op-rs/op-reth#402) Fixes a potential DoS vector of overflowing the stack by writing zero values to storage. Resolves op-rs/op-reth#401, op-rs/op-reth#394 * fix: store block updates metric fixed (op-rs/op-reth#409) Closes op-rs/op-reth#410 * feat: implemented `OpProofStorage` Database metrics (op-rs/op-reth#407) Closes op-rs/op-reth#224 Closes op-rs/op-reth#387 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: add `initialize-op-proofs` command (op-rs/op-reth#377) Adds `initialize-op-proofs` which is now required before running the node with the proof ExEx. Fixes op-rs/op-reth#376 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: integrate `initialize-op-proofs` to devnet (op-rs/op-reth#412) Closes op-rs/op-reth#411 --------- Co-authored-by: Julian Meyer <julian.meyer@coinbase.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: dashboard for proof history related metrics (op-rs/op-reth#414) Closes op-rs/op-reth#413 Closes op-rs/op-reth#405 The new dashboard looks like following: <img width="2199" height="1086" alt="Screenshot 2025-11-27 at 18 19 36" src="https://github.com/user-attachments/assets/8d8f4f32-f1c3-4704-8ee4-ed67921ae039" /> <img width="2199" height="959" alt="Screenshot 2025-11-27 at 18 20 04" src="https://github.com/user-attachments/assets/33714ecc-5853-4a19-a519-4a35efff5cd9" /> --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * fix: update metric on startup (op-rs/op-reth#419) Closes op-rs/op-reth#416 Issue: The issue was that CLI command `initialize-op-proofs` set the earliest block number but when the node starts, the metrics initialized are at default values. All other metrics get called soon after the start but not the earliest block number. Solution: Set the metric on startup, if available. * chore: code coverage for historical proof exex (op-rs/op-reth#415) Closes op-rs/op-reth#406 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: Add test for Display impl for PrunerOutput (op-rs/op-reth#435) Fixes op-rs/op-reth#432 * feat: add exex sync status in grafana (op-rs/op-reth#429) Closes op-rs/op-reth#426 <img width="1661" height="422" alt="Screenshot 2025-12-03 at 10 59 58" src="https://github.com/user-attachments/assets/91aeb374-a40a-45db-9c38-f9f36215747e" /> * feat: Consume TrieUpdates and HashedState from notification (op-rs/op-reth#439) ### Description This PR consumes trie updates and hashed state directly from notifications, falling back to full block execution when notification data is missing. ### Current limitations: There are known serialization/deserialization issues with the WAL. Before starting the node, ensure the WAL is cleaned. The Proof History ExEx is designed to tolerate missing notification data, but if you are running any additional ExEx components, ensure they can also safely handle gaps in notification data. Due to the above limitation, the resyncing test has been disabled. Closes op-rs/op-reth#420 Based on op-rs/op-reth#425 --------- Co-authored-by: Brian Picciano <me@mediocregopher.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: fall back to block execution if state updates missing from notification (op-rs/op-reth#455) Closes op-rs/op-reth#453 * feat: Implemented `OpProofStoragePrunerTask` (op-rs/op-reth#375) Closes op-rs/op-reth#361 Closes op-rs/op-reth#395 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: implemented pruner metrics (op-rs/op-reth#454) Closes op-rs/op-reth#363 * fix: new earliest block after pruning (op-rs/op-reth#470) Fixes: op-rs/op-reth#469 * chore(exex): Add metrics feature in `reth-optimism-exex` (op-rs/op-reth#438) Closes op-rs/op-reth#427 --------- Co-authored-by: itschaindev <jagrutk@protonmail.com> * fix(trie): Add missing `ValueWithSubkey` impl for `VersionedValue<T>` (op-rs/op-reth#475) Closes op-rs/op-reth#474 * chore: add prune e2e test back (op-rs/op-reth#477) Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: sysgo integration (op-rs/op-reth#478) Closes op-rs/op-reth#297 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: enable code coverage for e2e (op-rs/op-reth#485) Closes op-rs/op-reth#442 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore(chain-state): Receive sorted updates to exex (op-rs/op-reth#464) Closes op-rs/op-reth#460 Makes use of the sorted updates which are now already available in the `ExecutedBlock`, instead of sorting out removed accounts/slots later in `reth-optimism-trie` --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat(trie): Cover error branches of `execute_and_store_block_updates` (op-rs/op-reth#437) Fixes op-rs/op-reth#436 * feat: unwind command added (op-rs/op-reth#499) Closes op-rs/op-reth#451 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore(test): use `reth_optimism_chainspec::BASE_SEPOLIA` in tests (op-rs/op-reth#505) Update to use `reth_optimism_chainspec::BASE_SEPOLIA` in engine tests, now that the jovian timestamp is known * feat: prune cli added (op-rs/op-reth#507) Closes op-rs/op-reth#452 * test: add unit test for OpProofsStateProviderRef Debug impl (op-rs/op-reth#493) Adds unit test for the `Debug` implementation of `OpProofsStateProviderRef`. Closes op-rs/op-reth#434 --------- Co-authored-by: Emilia Hane <emiliaha95@gmail.com> * test: extended prune e2e with `getProof` consistency (op-rs/op-reth#514) Closes op-rs/op-reth#461 * feat: used `OpProofsStorageWithMetrics` provider on pruner (op-rs/op-reth#498) Closes op-rs/op-reth#472 * feat: pruning progress and metrics improvements (op-rs/op-reth#504) Closes op-rs/op-reth#362 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: implement prune batching (op-rs/op-reth#530) Closes op-rs/op-reth#364 * feat: address lookup table (op-rs/op-reth#509) Closes op-rs/op-reth#486 * feat: backfill address map (op-rs/op-reth#512) Closes op-rs/op-reth#510 * fix(trie): Fix broken `Debug` impl unit test (op-rs/op-reth#539) Fix broken unit test for verifying `Debug` impl, add missing map `address_mappings` to expected test output, introduced in op-rs/op-reth#512 * refactor(trie): return `OpProofsStorageError` from `store_block_updates` (op-rs/op-reth#537) Closes op-rs/op-reth#524 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: metrics panel for debug api ext (op-rs/op-reth#533) Closes op-rs/op-reth#515 Closes op-rs/op-reth#516 <img width="1607" height="277" alt="Screenshot 2025-12-18 at 20 44 35" src="https://github.com/user-attachments/assets/d135ff3c-c837-4dc8-a097-b9f818a08a35" /> --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * refactor(trie): return `OpProofsStorageError` from `execute_and_store_block_updates` (op-rs/op-reth#535) Closes op-rs/op-reth#523 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: initialize-op-proofs needs chain parameter (op-rs/op-reth#544) * fix: support Send in test_op_proofs_state_provider_ref_debug (op-rs/op-reth#546) * chore(trie): return `OpProofsStorageError` from `unwind_history` (op-rs/op-reth#548) Cherry picks commit from op-rs/op-reth#540 Co-authored-by: Himess <semihcvlk53@gmail.com> * refactor(trie): return OpProofsStorageError from unwind_and_store_block_updates (op-rs/op-reth#541) Closes op-rs/op-reth#525 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: integrated storing address mapping (op-rs/op-reth#534) Closes op-rs/op-reth#487 * chore: refactor op-proofs cli (op-rs/op-reth#554) Closes op-rs/op-reth#511 * feat: added startup safety check for massive pruning operation (op-rs/op-reth#556) Closes op-rs/op-reth#555 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: fix reorg e2e (op-rs/op-reth#558) Attempts to fix op-rs/op-reth#528 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: improve `append_or_delete_dup_sorted` (op-rs/op-reth#552) Closes op-rs/op-reth#501 * chore(trie): replace `eyre` with `OpProofsStorageError` (op-rs/op-reth#564) Closes op-rs/op-reth#527 * chore: revert hybrid storage experiment (op-rs/op-reth#568) Revert all the changes made as part of the hybrid storage experiment. * refactor(trie): extract save_hashed_accounts closure to method1 (op-rs/op-reth#570) Closes op-rs/op-reth#566 (1/5) Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: optimized history deletions (op-rs/op-reth#565) Closes op-rs/op-reth#311 Performace comparison under load test: ``` previous: 4.873750666s sorted delete: 3.278444083s sorted delete + seperate read (current as per the PR): 2.980065583s ``` * ci: restrict codecov workflows to merge_group and workflow_dispatch (op-rs/op-reth#513) ## Summary - Remove `push` and `pull_request` triggers from coverage workflows - Add `merge_group` trigger to run codecov only in merge queue - Keep `workflow_dispatch` for manual triggering when needed This reduces CI resource usage by running codecov only when necessary. Closes op-rs/op-reth#500 * feat: optimised initial state update for pruning (op-rs/op-reth#563) Closes op-rs/op-reth#562 Validated the optimization by rewriting an initial state with N=50,000 updated entries. - Throughput: Increased by ~20.9% (326k → 394k entries/sec). - Latency: Reduced by ~17.2% (153ms → 126ms). ``` Old approach: Rewrite Duration: 153.26ms Throughput: 326,248 entries/sec New approach: Rewrite Duration: 126.77ms Throughput: 394,410 entries/sec ``` bench: op-rs/op-reth@274e544 * docs: OpProofStorage DB Schema (op-rs/op-reth#583) Closes op-rs/op-reth#240 * chore: push devstack support for op-reth upstream (op-rs/op-reth#585) Part of op-rs/op-reth#483. First wait for theo's PR [op-rs/op-reth#18754](#18754) to be merged. Then wait for our PR [op-rs/op-reth#18772](#18772) to be merged. After that, in this repo, remove the current submodule and replace it with optimism itself. * feat: add verification interval for integrity check (op-rs/op-reth#577) Closes op-rs/op-reth#449 The approach used is to perform full block verification after every N blocks to ensure the state is still correct. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: backfill refactor (op-rs/op-reth#578) Closes op-rs/op-reth#566 --------- Co-authored-by: Himess <95512809+Himess@users.noreply.github.com> Co-authored-by: Himess <semihcvlk53@gmail.com> * chore: update optimism (op-rs/op-reth#590) * chore(tire): Improve test coverage for `OpProofsStorageError` (op-rs/op-reth#589) Ref op-rs/op-reth#588 Adds tests for conversions to `OpProofsStorageError` * fix: db_stat after upstream pull (op-rs/op-reth#601) * test: unit test for exex (op-rs/op-reth#599) Closes op-rs/op-reth#458 * fix: `MdbxStorageCursor` non first key issue (op-rs/op-reth#602) Closes op-rs/op-reth#600 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Sadiqur Rahman <sadiqurr8@gmail.com> * chore: added exex missing tests (op-rs/op-reth#605) Closes op-rs/op-reth#588 * chore: getProof benchmark utility added (op-rs/op-reth#550) Utility for op-rs/op-reth#446 * feat: refactor and optimise pruning (op-rs/op-reth#587) Closes op-rs/op-reth#567 This PR improves the performance of the historical state pruning logic. By refining how state updates are handled and written during pruning, we observe a significant reduction in write latency. Performance Benchmarks: Comparison run using [load_test](https://github.com/op-rs/op-reth/blob/8f0b61039a1af5f07793d5ea38e21e0439b7f0b3/crates/optimism/trie/src/db/store.rs#L3462) Metric | Current (Baseline) | Optimised (This PR) | Improvement -- | -- | -- | -- Write Time | ~107ms | ~50ms | ~2.1x Faster Total Duration | ~129ms | ~81ms | ~1.6x Faster * fix: prune redundant tombstones at history horizon (op-rs/op-reth#620) Closes op-rs/op-reth#619 * feat: backfill job restart-safe and idempotent (op-rs/op-reth#594) Closes op-rs/op-reth#244 * chore(trie): Use `reth-trie-common` to enable `serde-bincode-compat` in v1.10.1 (op-rs/op-reth#617) Ref op-rs/op-reth#611 Uses `reth-trie-common` dep explicitly in `reth-optimism-trie` to make sure `serde-bincode-compat` can be enabled while using reth release v1.10.1 * feat: backfill renamed to initialize (op-rs/op-reth#622) Closes op-rs/op-reth#606 * feat: refactored store (op-rs/op-reth#623) Closes op-rs/op-reth#626 * chore: mv proof args to rollup node (op-rs/op-reth#625) Closes op-rs/op-reth#613 * chore(trie): Revert redundant diff (op-rs/op-reth#610) Ref op-rs/op-reth#611 Uses getters instead of direct field access for trie updates to remove redundant diff with upstream * chore: use reth versioned deps (op-rs/op-reth#627) Closes op-rs/op-reth#621 This PR updates the workspace configuration to use reth as a git dependency instead of a local crate, including necessary fixes for the compiler, CI workflows, and test configurations. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore(trie): Fix lint (op-rs/op-reth#636) Lint fix. Clippy is catching since yesterday's new nightly release. * fix: removed unused pruner metrics (op-rs/op-reth#638) Closes op-rs/op-reth#607 * chore(trie): Remove redundant generic lifetime for `InitializationJob` (op-rs/op-reth#632) Ref op-rs/op-reth#631 - Changes nested database tx type to owned type as has no need to be ref - Uses `AsyncFnOnce`(new in Rust 1.85) instead of generic future return type * chore(trie): Make `initialize` a method of `InitializationJob` (op-rs/op-reth#635) Ref op-rs/op-reth#631 - Removes `safe_fn` closure as arg from `initialize` by making `initialize` a method of `InitializationJob` - Moves save functions out of `InitializationJob` into impls of trait method of new trait `InitTable::save_entries` - Uses `size_hint` to avoid cost of heap re-allocations - Removes redundant clone * refactor(trie): use parking_lot::RwLock in InMemoryProofsStorage (op-rs/op-reth#637) Closes op-rs/op-reth#633 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: async notification processing (op-rs/op-reth#614) This PR improves the robustness of the OP-Reth Proofs ExEx by process notifications asynchronously(to avoid WAL backpressure) when the gap between proof storage and the tip is big. This change ensures that notification pipeline is able to keep up and gracefully handle the gap. Closes op-rs/op-reth#597 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: simplify OpProofsStore (op-rs/op-reth#639) Closes: op-rs/op-reth#634 Blocked on: op-rs/op-reth#637 --------- Co-authored-by: Himess <semihcvlk53@gmail.com> * chore: ExEx config builder (op-rs/op-reth#642) Closes op-rs/op-reth#641 --------- Co-authored-by: Himess <semihcvlk53@gmail.com> * chore(trie): Fix redundant type annotations (op-rs/op-reth#645) Fixes lint `clippy::redundant_type_annotations` * chore(trie): Replace redundant clone of entries in `InitializationJob` (op-rs/op-reth#644) Closes op-rs/op-reth#631 - Puts off assigning variables until right when needed - Optimizes away redundant clone of entries on batch insertion - Updates symbol name to better show initialising is a batching algorithm * chore: moved proof initialization to `reth-optimism-node` (op-rs/op-reth#640) Closes op-rs/op-reth#612 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: separate initial state store trait (op-rs/op-reth#647) Closes op-rs/op-reth#643 * chore(test): support reth as in sequencer and validator (op-rs/op-reth#650) This PR added support to run op-reth in sequencer and op-reth(with proof) in validator for the e2e test. Note: In this setup pruning test isn't supported(at this moment) since this test is falling outside the proof window of reth. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: fix flaky e2e test (op-rs/op-reth#653) Closes op-rs/op-reth#528 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore(test): dep dump (op-rs/op-reth#654) Co-authored-by: Sadiqur Rahman <sadiqurr8@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Himess <95512809+Himess@users.noreply.github.com> Co-authored-by: Himess <semihcvlk53@gmail.com> Co-authored-by: jagroot <4516586+itschaindev@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Julian Meyer <julianmeyer2000@gmail.com> * chore(op-reth/historical-proofs): fix compilation errors --------- Signed-off-by: Julian Meyer <julian.meyer@coinbase.com> Co-authored-by: Georgios Konstantopoulos <me@gakonst.com> Co-authored-by: Amp <amp@ampcode.com> Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com> Co-authored-by: Xzavier <xingqiang.yuan@okg.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Julian Meyer <julianmeyer2000@gmail.com> Co-authored-by: jagroot <4516586+itschaindev@users.noreply.github.com> Co-authored-by: Varun Doshi <doshivarun202@gmail.com> Co-authored-by: Sadiqur Rahman <sadiqurr8@gmail.com> Co-authored-by: einar-oplabs <oplabs@einar.io> Co-authored-by: Einar Rasmussen <einar@oplabs.co> Co-authored-by: op-will <232669456+op-will@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Emilia Hane <emiliaha95@gmail.com> Co-authored-by: itschaindev <jagrutk@protonmail.com> Co-authored-by: Jonas Warlike <mikmillarog@protonmail.com> Co-authored-by: Julian Meyer <julian.meyer@coinbase.com> Co-authored-by: Brian Picciano <me@mediocregopher.com> Co-authored-by: Himess <95512809+Himess@users.noreply.github.com> Co-authored-by: Himess <semihcvlk53@gmail.com> Co-authored-by: Sebastian Stammler <seb@oplabs.co>

Fixes op-rs/op-reth#246 - Return `UpdateResult` from `store_trie_updates` with counts of items stored. - Log counts and latencies for live collector Signed-off-by: Julian Meyer <julian.meyer@coinbase.com>

* feat(rust): unify workspaces (#19034) * chore(rust): merge all the rust workspaces * chore(ci): formatting, fixing dockerfile and ci * chore: more ci fixes * Update .circleci/continue/rust-ci.yml Co-authored-by: Sebastian Stammler <seb@oplabs.co> * Update .circleci/continue/rust-e2e.yml Co-authored-by: Sebastian Stammler <seb@oplabs.co> * Update .circleci/continue/rust-e2e.yml Co-authored-by: Sebastian Stammler <seb@oplabs.co> * chore: pr comments * nit: fix kona path in op-challenger * chore: fix prestate artifacts path --------- Co-authored-by: Sebastian Stammler <seb@oplabs.co> * chore(op-reth/docs): port docs to op-reth (#18968) * chore(kona/batcher): remove unused batcher crate (#18972) * chore(rust): consolidate check-no-std recipes into root justfile (#19154) * kona: build cannon from local monorepo source instead of cloning a pinned tag (#19183) Now that kona lives in the monorepo, the prestate build no longer needs to clone the optimism repo at a pinned commit to build cannon. Instead, cannon is built from the local monorepo source via a Docker named build context, removing the cannon_tag config file and simplifying the build. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * proofs: port TestInteropFaultProofs action test to devstack (#19163) * proofs: port TestInteropFaultProofs action test to devstack * fix(kona/client): produce InvalidTransition when derivation falls short of target block * Update Makefile Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com> * fix makefile indent * revert Makefile changes * revert rust/kona/justfile changes --------- Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com> * chore(rust): unify rust docs and update repo links (#19101) * chore(rust): move docs to unified rust/docs/ directory * chore(rust): update repo links from alloy-rs/paradigmxyz to ethereum-optimism * feat(release/rust): set up release process for rust crates (#19155) - Add tag classification to distinguish Rust crate groups from Docker-only groups - Add publish-crates CI job for crates.io publishing with dependency-ordered releases - Add op-reth Docker bake target and compute-git-versions entry - Add cargo-release configuration for alloy-op-hardforks and op-reth - Add .dockerignore for rust/ build context - Fix op-reth DockerfileOp manifest paths * refactor(db): derive Clone for DatabaseEnv (paradigmxyz/reth#21641) Co-authored-by: Amp <amp@ampcode.com> * chore(deps): breaking bumps (paradigmxyz/reth#21584) Co-authored-by: Georgios Konstantopoulos <me@gakonst.com> Co-authored-by: Amp <amp@ampcode.com> * refactor: add with_* compressor utility methods (paradigmxyz/reth#21680) * refactor: use alloy_primitives::map for all HashMap/HashSet types (paradigmxyz/reth#21686) Co-authored-by: Amp <amp@ampcode.com> * fix: support EIP-1559 params configuration for Optimism dev mode (paradigmxyz/reth#21855) * chore(rust/op-reth): pull latest changes and update dependencies * chore(op-exex): Add crate `reth-optimism-exex` and `reth-optimism-trie` (op-rs/op-reth#204) - Adds new crates `reth-optimism-exex` and `reth_optimism_trie` - Moves `reth_exex::external_proofs` -> `reth_optimism_exex` - Moves `reth_exex::external_proofs::storage` -> `reth_optimism_trie` * fix: ci lint and doc (op-rs/op-reth#234) Seems like when `reth-optimism-trie` depends on `reth-db-api` indirectly, the doctest crate couldn’t resolve `reth_db_api::…` and failing. Closes op-rs/op-reth#233 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: separate storage and account cursors (op-rs/op-reth#229) This PR separates storage and account cursors which makes sense because these two cursors may have very different seek/next implementations. In the case of in-memory, a single cursor impl can still handle both, but for MDBX, it makes sense to implement separate cursors. Closes op-rs/op-reth#236 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: add external state provider implementation (op-rs/op-reth#197) Based on https://github.com/op-rs/op-reth/pull/203 , https://github.com/op-rs/op-reth/pull/204 This PR implements `StateProvider` given a `OpProofsStorage` instance. It reads most data from the external database, but falls back on the latest provider for block hashes and code by hash similar to the existing historical provider in Reth. This is an important part to implementing live syncing since we're running the sync process on the DB being created. In the Reth implementation, `proof.rs` is contained in `trie/db`, so I think it makes sense to go in our DB crate. The `provider.rs` is in the `reth-provider` crate, but I don't think a separate provider crate helps us here, so I think we should also include that in the trie crate as well. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: add deletion tests for op proofs storage (op-rs/op-reth#230) Based on op-rs/op-reth#229 Adds tests to ensure that storing TrieUpdates that include deletions actually deletes the nodes at that block height, and that updates take precedence over deletions (same as `write_trie_updates` in Reth). Closes op-rs/op-reth#238 * feat: implement live state collector using external provider (op-rs/op-reth#198) Based on https://github.com/op-rs/op-reth/pull/197 This PR implements the live state collector on top of the `OpProofsStateProvider` created in the previous PR. This sync process tries to re-execute all blocks from the current block of the external database to tip. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: add binary entry point for external proofs in OP (op-rs/op-reth#222) Closes op-rs/op-reth#164 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: store hashed account and storage (op-rs/op-reth#235) Closes op-rs/op-reth#211 * feat: implement cursor read/write metrics (op-rs/op-reth#232) Fixes op-rs/op-reth#225 (does not track errors as these are fatal currently) * feat: store account and storage branches (op-rs/op-reth#255) Closes op-rs/op-reth#210 * feat: Concrete error variants for `reth-optimism-trie` (op-rs/op-reth#251) Fixes op-rs/op-reth#237 Adds more concrete error variants for `OpProofsStorageError` Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: Remove block_number from initial sync store methods (op-rs/op-reth#248) Fixes op-rs/op-reth#247 Removes the `block_number` parameter from: - store_account_branches - store_storage_branches - store_hashed_accounts - store_hashed_storages Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat(MDBX): Implement earliest and latest block tracking in storage (op-rs/op-reth#253) Fixes op-rs/op-reth#215 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat(deps): Add `serde-bincode-compat` feature to `reth-trie` (op-rs/op-reth#250) Ref https://github.com/op-rs/op-reth/issues/241 - Fixes comment from deps fix in https://github.com/op-rs/op-reth/pull/229 (fix correct but comment misleading) - `serde-bincode-compat` is a feature which must be enabled when the `bincode` dependency is used. Asides for dev-deps, this only happens in `reth-stages`. `reth-exex` runs in `reth-stages`. In this wokrstream we use the `TrieUpdate` in exex. `serde-bincode-compat` is already implemented for `TrieUpdates` in `reth-trie-common`, this PR simply makes that feature accessible via `reth-trie` and in turn `reth-optimism-trie`, in order to enable it in `reth-optimism-exex`. * feat: store trie updates (op-rs/op-reth#258) Closes: op-rs/op-reth#214 * feat: implemented mdbx trie cursor (op-rs/op-reth#249) Closes op-rs/op-reth#212 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore(deps): Rm redundant dep (op-rs/op-reth#260) Ref https://github.com/op-rs/op-reth/issues/205 Removes redundant dep `reth-db-api` from `reth-optimism-trie`, since used types are re-exported via `reth-db` --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * refactor: replace tuple with `BlockNumHash` (op-rs/op-reth#254) Resolves op-rs/op-reth#239 --------- Co-authored-by: Einar Rasmussen <einar@oplabs.co> * feat: Return ref to B256 (op-rs/op-reth#272) Fixes op-rs/op-reth#270 * chore(exex): Add missing crate header to `reth-optimism-exex` (op-rs/op-reth#271) Resolves op-rs/op-reth#243 --------- Co-authored-by: Einar Rasmussen <einar@oplabs.co> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: implemented hashed cursors (op-rs/op-reth#268) Closes op-rs/op-reth#213 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: add support for `eth_getProof` (op-rs/op-reth#257) Adds support for `eth_getProof` RPC method. This required reworking the launch command to still work with an alternative DB provider. Closes op-rs/op-reth#173 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: implement proof collection e2e tests (op-rs/op-reth#231) Implements end to end tests for proof collection. These tests create a blockchain, run the backfill job, then run the live collector. The live collector executes on top of the backfilled state and it already checks that the state root matches, so the tests just ensure that it runs without errors. Fixes op-rs/op-reth#170 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * perf(trie): feature gate `reth-optimism-trie` metrics (op-rs/op-reth#282) Closes https://github.com/op-rs/op-reth/issues/281 - Feature gates `reth-optimism-trie` metrics - Moves cursor impls out of proofs module into new module `cursor` - Moves cursor factory impls into new module `cursor_factory` - Updates cursor factory impls to return cursor types with metrics wrapper if metrics feature is enabled * chore(docs): Add missing doclinks (op-rs/op-reth#277) Adds doc links to all types to make sure they are caught by lint if names of types change in the future * fix(docs): Fixes docs of `reth_optimism_trie::cursor_factory` (op-rs/op-reth#286) Fixes module docs of `reth_optimism_trie::cursor_factory`, which weren't updated on moving the code out of the proofs module * fix(test): Enable live collector tests with metrics feature (op-rs/op-reth#291) Closes https://github.com/op-rs/op-reth/issues/283 Enable live collector tests when metrics feature is enabled * fix(docs): Fixes doc links (op-rs/op-reth#292) Fixes some doc links from https://github.com/op-rs/op-reth/pull/277 Several places should have linked to `OpProofsStore` which linked to `OpProofsStorage` * feat: Track eth_getProof metrics (op-rs/op-reth#285) Adds latency, total requests, successful responses, and error responses metrics to the eth_getProof RPC method. closes op-rs/op-reth#226 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: devnet setup (op-rs/op-reth#295) Closes op-rs/op-reth#200 * feat: implement `debug_executePayload` (op-rs/op-reth#276) Closes https://github.com/op-rs/op-reth/issues/189 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: storage trie cursor (op-rs/op-reth#301) Closes op-rs/op-reth#299 * feat: latest block updates in `store_trie_updates` (op-rs/op-reth#304) Closes op-rs/op-reth#302 * feat: live collector integration (op-rs/op-reth#306) Closes op-rs/op-reth#296 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: add mdbx to storage tests (op-rs/op-reth#308) Fixes a small issue where storage slots with a zero value weren't skipped. I implemented this above the BlockVersionedCursor since it only applies to the storage cursor. Also enables MDBX in the common storage tests and ensures that no storage tests insert block numbers out of order or insert duplicate keys. * fix: merge conflict caused by storage tests (op-rs/op-reth#313) Fix merge conflicts caused by storage API change. * feat: create index table to efficiently prune history tables (op-rs/op-reth#280) Closes op-rs/op-reth#219 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Julian Meyer <julianmeyer2000@gmail.com> * chore(trie): Move `OpProofsStorageError` to own module (op-rs/op-reth#317) Ref https://github.com/op-rs/op-reth/issues/316 Moves `OpProofsStorageError` into own module for readability Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: Named fields for OpProofsStorageError variants (op-rs/op-reth#319) Fixes op-rs/op-reth#315 --------- Co-authored-by: Emilia Hane <emiliaha95@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: implement `debug_executionWitness` (op-rs/op-reth#312) Add support for `debug_executionWitness` API. This hasn't been tested, but I'll be able to test it soon once the backfill job completes. This code is mostly copied from here and adapted to work with our state provider: https://github.com/op-rs/op-reth/blob/1470a9cc77af798c999da05d9541a7025e5d4924/crates/rpc/rpc/src/debug.rs#L643-L695 Fixes op-rs/op-reth#190 * chore(trie): Rename metrics scope (op-rs/op-reth#322) Ref https://github.com/op-rs/op-reth/issues/321 Renames scope for `BlockMetrics` and `OperationMetrics` from `external_proofs` to `optimism_trie` * feat: implemented wiped storage handling on `store_trie_updates` (op-rs/op-reth#300) Closes op-rs/op-reth#259 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore(trie): Rename `OpProofsStorageError` variant (op-rs/op-reth#324) Ref https://github.com/op-rs/op-reth/issues/315 Renames `OpProofsStorageError::BlockUpdateFailed` to `OpProofsStorageError::MissingParentBlock` which better matches the error message Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * fix(db): `is_parent check` in `store_trie_update` (op-rs/op-reth#309) Fixes op-rs/op-reth#305 * feat: fetch trie updates (op-rs/op-reth#327) Closes: op-rs/op-reth#256 --------- Co-authored-by: itschaindev <jagrutk@protonmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: jagroot <4516586+itschaindev@users.noreply.github.com> * feat: implemented mdbx `replace_updates` (op-rs/op-reth#329) Closes op-rs/op-reth#193 Closes op-rs/op-reth#216 * feat: apply `BlockStateDiff` to the `prune_earliest_state` method (op-rs/op-reth#332) Closes op-rs/op-reth#217 Also fixed a bug where the `set_earliest_block_number_hash` was using `append` instead of `upsert`. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: refactor storage trie cursor (op-rs/op-reth#338) Closes op-rs/op-reth#337 * feat: integrate metrics into live sync process (op-rs/op-reth#330) Fixes op-rs/op-reth#246 - Return `UpdateResult` from `store_trie_updates` with counts of items stored. - Log counts and latencies for live collector Signed-off-by: Julian Meyer <julian.meyer@coinbase.com> * test: basic storage proof (op-rs/op-reth#336) Part 1 for op-rs/op-reth#289 * test: `eth_getProof` for account proof validation (op-rs/op-reth#341) Closes op-rs/op-reth#288 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * test: `debug_executePayload` RPC (op-rs/op-reth#340) Closes op-rs/op-reth#325 * test: multi slot and zero testing (op-rs/op-reth#343) Closes op-rs/op-reth#289 * test: resyncing (op-rs/op-reth#350) Closes op-rs/op-reth#349 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * test: e2e for `debug_executionWitness` (op-rs/op-reth#348) Closes op-rs/op-reth#326 Fixed the `nonce too low` issue with the way `execution_witness` was implemented for op. * test: doc test eg for exex setup (op-rs/op-reth#352) Closes op-rs/op-reth#342 * chore: enable remaining mdbx tests (op-rs/op-reth#354) Fixes op-rs/op-reth#252 * fix: ensure db tx is closed after backfill (op-rs/op-reth#357) The db tx isn't dropped ever for the backfill job, so create a temp scope to ensure it's dropped. * refactor: `MdbxProofStorage` (op-rs/op-reth#351) Closes op-rs/op-reth#335 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * fix: rebase conflicts (op-rs/op-reth#367) Fix conflicts rebasing onto latest upstream main * feat: handle reorg (op-rs/op-reth#356) Closes op-rs/op-reth#191 --------- * feat: add target debug logs to reth_optimism_exex (op-rs/op-reth#368) Fixes op-rs/op-reth#366 * test: basic reorg (op-rs/op-reth#369) Part of op-rs/op-reth#174 * fix: missing block after reorg (op-rs/op-reth#382) Closes op-rs/op-reth#379 * feat: add sync status RPC (op-rs/op-reth#353) Adds `debug_proofsSyncStatus` for fetching the proof window when enabled. Also adds metrics for the earliest and latest block numbers available. * feat: replace variant OpProofsStorageError::Other (op-rs/op-reth#355) Fixes op-rs/op-reth#316 I also noticed a few `DatabaseError` that were being mapped to `OpProofsStorageError::Other` and changed them to point to `OpProofsStorageError::DatabaseError` --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: Implemented `OpProofStoragePruner` (op-rs/op-reth#360) Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: handle `ChainReverted` notification (op-rs/op-reth#380) Closes op-rs/op-reth#192 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: small lint (op-rs/op-reth#390) Find a small typo here. * test: reorg test scenarios (op-rs/op-reth#384) Closes op-rs/op-reth#174 * test: add new devnet for op-reth as sequencer (op-rs/op-reth#391) Closes op-rs/op-reth#389 * chore(trie): Simplify cursor interface (op-rs/op-reth#388) Ref https://github.com/op-rs/op-reth/issues/371 Removes redundant `OpProofsTrieCursorRO` and `OpProofsHashedCursorRO` traits by enabling a `DatabaseError` variant that wraps an error trait object. This way we can have a bijective relation between `DatabaseError` and `OpProofsStorageError` in a type safe way, i.e. conversion both ways without matching on string. cc @dhyaniarun1993 * feat: add panels for `eth_getProof` in grafana dashboard (op-rs/op-reth#392) Closes op-rs/op-reth#318 <img width="2208" height="672" alt="Screenshot 2025-11-21 at 15 35 16" src="https://github.com/user-attachments/assets/62ba3b36-fd4f-498a-9bbc-7e4f33c9e3d2" /> The OP-Reth dashboard is not loaded by default when running the kurtosis devnet for e2e historical proofs. Once op-rs/op-reth#391 is merged, I'll update the devnet config to load it by default. * fix: add custom TrieType to separate metrics for op proofs storage (op-rs/op-reth#397) Closes op-rs/op-reth#321 * feat: implement methods for new traits for `inmemory` cursor (op-rs/op-reth#396) Closes op-rs/op-reth#400 * fix: use loop instead of recursing in hashed next (op-rs/op-reth#402) Fixes a potential DoS vector of overflowing the stack by writing zero values to storage. Resolves op-rs/op-reth#401, op-rs/op-reth#394 * fix: store block updates metric fixed (op-rs/op-reth#409) Closes op-rs/op-reth#410 * feat: implemented `OpProofStorage` Database metrics (op-rs/op-reth#407) Closes op-rs/op-reth#224 Closes op-rs/op-reth#387 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: add `initialize-op-proofs` command (op-rs/op-reth#377) Adds `initialize-op-proofs` which is now required before running the node with the proof ExEx. Fixes op-rs/op-reth#376 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: integrate `initialize-op-proofs` to devnet (op-rs/op-reth#412) Closes op-rs/op-reth#411 --------- Co-authored-by: Julian Meyer <julian.meyer@coinbase.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: dashboard for proof history related metrics (op-rs/op-reth#414) Closes op-rs/op-reth#413 Closes op-rs/op-reth#405 The new dashboard looks like following: <img width="2199" height="1086" alt="Screenshot 2025-11-27 at 18 19 36" src="https://github.com/user-attachments/assets/8d8f4f32-f1c3-4704-8ee4-ed67921ae039" /> <img width="2199" height="959" alt="Screenshot 2025-11-27 at 18 20 04" src="https://github.com/user-attachments/assets/33714ecc-5853-4a19-a519-4a35efff5cd9" /> --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * fix: update metric on startup (op-rs/op-reth#419) Closes op-rs/op-reth#416 Issue: The issue was that CLI command `initialize-op-proofs` set the earliest block number but when the node starts, the metrics initialized are at default values. All other metrics get called soon after the start but not the earliest block number. Solution: Set the metric on startup, if available. * chore: code coverage for historical proof exex (op-rs/op-reth#415) Closes op-rs/op-reth#406 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: Add test for Display impl for PrunerOutput (op-rs/op-reth#435) Fixes op-rs/op-reth#432 * feat: add exex sync status in grafana (op-rs/op-reth#429) Closes op-rs/op-reth#426 <img width="1661" height="422" alt="Screenshot 2025-12-03 at 10 59 58" src="https://github.com/user-attachments/assets/91aeb374-a40a-45db-9c38-f9f36215747e" /> * feat: Consume TrieUpdates and HashedState from notification (op-rs/op-reth#439) ### Description This PR consumes trie updates and hashed state directly from notifications, falling back to full block execution when notification data is missing. ### Current limitations: There are known serialization/deserialization issues with the WAL. Before starting the node, ensure the WAL is cleaned. The Proof History ExEx is designed to tolerate missing notification data, but if you are running any additional ExEx components, ensure they can also safely handle gaps in notification data. Due to the above limitation, the resyncing test has been disabled. Closes op-rs/op-reth#420 Based on https://github.com/op-rs/op-reth/pull/425 --------- Co-authored-by: Brian Picciano <me@mediocregopher.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: fall back to block execution if state updates missing from notification (op-rs/op-reth#455) Closes op-rs/op-reth#453 * feat: Implemented `OpProofStoragePrunerTask` (op-rs/op-reth#375) Closes op-rs/op-reth#361 Closes op-rs/op-reth#395 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: implemented pruner metrics (op-rs/op-reth#454) Closes op-rs/op-reth#363 * fix: new earliest block after pruning (op-rs/op-reth#470) Fixes: op-rs/op-reth#469 * chore(exex): Add metrics feature in `reth-optimism-exex` (op-rs/op-reth#438) Closes https://github.com/op-rs/op-reth/issues/427 --------- Co-authored-by: itschaindev <jagrutk@protonmail.com> * fix(trie): Add missing `ValueWithSubkey` impl for `VersionedValue<T>` (op-rs/op-reth#475) Closes https://github.com/op-rs/op-reth/issues/474 * chore: add prune e2e test back (op-rs/op-reth#477) Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: sysgo integration (op-rs/op-reth#478) Closes op-rs/op-reth#297 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: enable code coverage for e2e (op-rs/op-reth#485) Closes op-rs/op-reth#442 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore(chain-state): Receive sorted updates to exex (op-rs/op-reth#464) Closes https://github.com/op-rs/op-reth/issues/460 Makes use of the sorted updates which are now already available in the `ExecutedBlock`, instead of sorting out removed accounts/slots later in `reth-optimism-trie` --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat(trie): Cover error branches of `execute_and_store_block_updates` (op-rs/op-reth#437) Fixes op-rs/op-reth#436 * feat: unwind command added (op-rs/op-reth#499) Closes op-rs/op-reth#451 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore(test): use `reth_optimism_chainspec::BASE_SEPOLIA` in tests (op-rs/op-reth#505) Update to use `reth_optimism_chainspec::BASE_SEPOLIA` in engine tests, now that the jovian timestamp is known * feat: prune cli added (op-rs/op-reth#507) Closes op-rs/op-reth#452 * test: add unit test for OpProofsStateProviderRef Debug impl (op-rs/op-reth#493) Adds unit test for the `Debug` implementation of `OpProofsStateProviderRef`. Closes op-rs/op-reth#434 --------- Co-authored-by: Emilia Hane <emiliaha95@gmail.com> * test: extended prune e2e with `getProof` consistency (op-rs/op-reth#514) Closes op-rs/op-reth#461 * feat: used `OpProofsStorageWithMetrics` provider on pruner (op-rs/op-reth#498) Closes op-rs/op-reth#472 * feat: pruning progress and metrics improvements (op-rs/op-reth#504) Closes op-rs/op-reth#362 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: implement prune batching (op-rs/op-reth#530) Closes op-rs/op-reth#364 * feat: address lookup table (op-rs/op-reth#509) Closes op-rs/op-reth#486 * feat: backfill address map (op-rs/op-reth#512) Closes op-rs/op-reth#510 * fix(trie): Fix broken `Debug` impl unit test (op-rs/op-reth#539) Fix broken unit test for verifying `Debug` impl, add missing map `address_mappings` to expected test output, introduced in https://github.com/op-rs/op-reth/pull/512 * refactor(trie): return `OpProofsStorageError` from `store_block_updates` (op-rs/op-reth#537) Closes op-rs/op-reth#524 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: metrics panel for debug api ext (op-rs/op-reth#533) Closes op-rs/op-reth#515 Closes op-rs/op-reth#516 <img width="1607" height="277" alt="Screenshot 2025-12-18 at 20 44 35" src="https://github.com/user-attachments/assets/d135ff3c-c837-4dc8-a097-b9f818a08a35" /> --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * refactor(trie): return `OpProofsStorageError` from `execute_and_store_block_updates` (op-rs/op-reth#535) Closes op-rs/op-reth#523 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: initialize-op-proofs needs chain parameter (op-rs/op-reth#544) * fix: support Send in test_op_proofs_state_provider_ref_debug (op-rs/op-reth#546) * chore(trie): return `OpProofsStorageError` from `unwind_history` (op-rs/op-reth#548) Cherry picks commit from https://github.com/op-rs/op-reth/pull/540 Co-authored-by: Himess <semihcvlk53@gmail.com> * refactor(trie): return OpProofsStorageError from unwind_and_store_block_updates (op-rs/op-reth#541) Closes op-rs/op-reth#525 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: integrated storing address mapping (op-rs/op-reth#534) Closes op-rs/op-reth#487 * chore: refactor op-proofs cli (op-rs/op-reth#554) Closes op-rs/op-reth#511 * feat: added startup safety check for massive pruning operation (op-rs/op-reth#556) Closes op-rs/op-reth#555 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: fix reorg e2e (op-rs/op-reth#558) Attempts to fix op-rs/op-reth#528 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: improve `append_or_delete_dup_sorted` (op-rs/op-reth#552) Closes op-rs/op-reth#501 * chore(trie): replace `eyre` with `OpProofsStorageError` (op-rs/op-reth#564) Closes op-rs/op-reth#527 * chore: revert hybrid storage experiment (op-rs/op-reth#568) Revert all the changes made as part of the hybrid storage experiment. * refactor(trie): extract save_hashed_accounts closure to method1 (op-rs/op-reth#570) Closes op-rs/op-reth#566 (1/5) Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: optimized history deletions (op-rs/op-reth#565) Closes op-rs/op-reth#311 Performace comparison under load test: ``` previous: 4.873750666s sorted delete: 3.278444083s sorted delete + seperate read (current as per the PR): 2.980065583s ``` * ci: restrict codecov workflows to merge_group and workflow_dispatch (op-rs/op-reth#513) ## Summary - Remove `push` and `pull_request` triggers from coverage workflows - Add `merge_group` trigger to run codecov only in merge queue - Keep `workflow_dispatch` for manual triggering when needed This reduces CI resource usage by running codecov only when necessary. Closes op-rs/op-reth#500 * feat: optimised initial state update for pruning (op-rs/op-reth#563) Closes op-rs/op-reth#562 Validated the optimization by rewriting an initial state with N=50,000 updated entries. - Throughput: Increased by ~20.9% (326k → 394k entries/sec). - Latency: Reduced by ~17.2% (153ms → 126ms). ``` Old approach: Rewrite Duration: 153.26ms Throughput: 326,248 entries/sec New approach: Rewrite Duration: 126.77ms Throughput: 394,410 entries/sec ``` bench: https://github.com/op-rs/op-reth/pull/563/changes/274e544adbcf7b2adfa70d4ba32ada916d5494d6 * docs: OpProofStorage DB Schema (op-rs/op-reth#583) Closes op-rs/op-reth#240 * chore: push devstack support for op-reth upstream (op-rs/op-reth#585) Part of op-rs/op-reth#483. First wait for theo's PR [op-rs/op-reth#18754](https://github.com/ethereum-optimism/optimism/pull/18754) to be merged. Then wait for our PR [op-rs/op-reth#18772](https://github.com/ethereum-optimism/optimism/pull/18772) to be merged. After that, in this repo, remove the current submodule and replace it with optimism itself. * feat: add verification interval for integrity check (op-rs/op-reth#577) Closes op-rs/op-reth#449 The approach used is to perform full block verification after every N blocks to ensure the state is still correct. --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: backfill refactor (op-rs/op-reth#578) Closes op-rs/op-reth#566 --------- Co-authored-by: Himess <95512809+Himess@users.noreply.github.com> Co-authored-by: Himess <semihcvlk53@gmail.com> * chore: update optimism (op-rs/op-reth#590) * chore(tire): Improve test coverage for `OpProofsStorageError` (op-rs/op-reth#589) Ref https://github.com/op-rs/op-reth/issues/588 Adds tests for conversions to `OpProofsStorageError` * fix: db_stat after upstream pull (op-rs/op-reth#601) * test: unit test for exex (op-rs/op-reth#599) Closes op-rs/op-reth#458 * fix: `MdbxStorageCursor` non first key issue (op-rs/op-reth#602) Closes op-rs/op-reth#600 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> Co-authored-by: Sadiqur Rahman <sadiqurr8@gmail.com> * chore: added exex missing tests (op-rs/op-reth#605) Closes op-rs/op-reth#588 * chore: getProof benchmark utility added (op-rs/op-reth#550) Utility for op-rs/op-reth#446 * feat: refactor and optimise pruning (op-rs/op-reth#587) Closes op-rs/op-reth#567 This PR improves the performance of the historical state pruning logic. By refining how state updates are handled and written during pruning, we observe a significant reduction in write latency. Performance Benchmarks: Comparison run using [load_test](https://github.com/op-rs/op-reth/blob/8f0b61039a1af5f07793d5ea38e21e0439b7f0b3/crates/optimism/trie/src/db/store.rs#L3462) Metric | Current (Baseline) | Optimised (This PR) | Improvement -- | -- | -- | -- Write Time | ~107ms | ~50ms | ~2.1x Faster Total Duration | ~129ms | ~81ms | ~1.6x Faster * fix: prune redundant tombstones at history horizon (op-rs/op-reth#620) Closes op-rs/op-reth#619 * feat: backfill job restart-safe and idempotent (op-rs/op-reth#594) Closes op-rs/op-reth#244 * chore(trie): Use `reth-trie-common` to enable `serde-bincode-compat` in v1.10.1 (op-rs/op-reth#617) Ref https://github.com/op-rs/op-reth/issues/611 Uses `reth-trie-common` dep explicitly in `reth-optimism-trie` to make sure `serde-bincode-compat` can be enabled while using reth release v1.10.1 * feat: backfill renamed to initialize (op-rs/op-reth#622) Closes op-rs/op-reth#606 * feat: refactored store (op-rs/op-reth#623) Closes op-rs/op-reth#626 * chore: mv proof args to rollup node (op-rs/op-reth#625) Closes op-rs/op-reth#613 * chore(trie): Revert redundant diff (op-rs/op-reth#610) Ref https://github.com/op-rs/op-reth/issues/611 Uses getters instead of direct field access for trie updates to remove redundant diff with upstream * chore: use reth versioned deps (op-rs/op-reth#627) Closes https://github.com/op-rs/op-reth/issues/621 This PR updates the workspace configuration to use reth as a git dependency instead of a local crate, including necessary fixes for the compiler, CI workflows, and test configurations. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore(trie): Fix lint (op-rs/op-reth#636) Lint fix. Clippy is catching since yesterday's new nightly release. * fix: removed unused pruner metrics (op-rs/op-reth#638) Closes op-rs/op-reth#607 * chore(trie): Remove redundant generic lifetime for `InitializationJob` (op-rs/op-reth#632) Ref https://github.com/op-rs/op-reth/issues/631 - Changes nested database tx type to owned type as has no need to be ref - Uses `AsyncFnOnce`(new in Rust 1.85) instead of generic future return type * chore(trie): Make `initialize` a method of `InitializationJob` (op-rs/op-reth#635) Ref https://github.com/op-rs/op-reth/issues/631 - Removes `safe_fn` closure as arg from `initialize` by making `initialize` a method of `InitializationJob` - Moves save functions out of `InitializationJob` into impls of trait method of new trait `InitTable::save_entries` - Uses `size_hint` to avoid cost of heap re-allocations - Removes redundant clone * refactor(trie): use parking_lot::RwLock in InMemoryProofsStorage (op-rs/op-reth#637) Closes op-rs/op-reth#633 Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: async notification processing (op-rs/op-reth#614) This PR improves the robustness of the OP-Reth Proofs ExEx by process notifications asynchronously(to avoid WAL backpressure) when the gap between proof storage and the tip is big. This change ensures that notification pipeline is able to keep up and gracefully handle the gap. Closes op-rs/op-reth#597 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * chore: simplify OpProofsStore (op-rs/op-reth#639) Closes: op-rs/op-reth#634 Blocked on: https://github.com/op-rs/op-reth/pull/637 --------- Co-authored-by: Himess <semihcvlk53@gmail.com> * chore: ExEx config builder (op-rs/op-reth#642) Closes op-rs/op-reth#641 --------- Co-authored-by: Himess <semihcvlk53@gmail.com> * chore(trie): Fix redundant type annotations (op-rs/op-reth#645) Fixes lint `clippy::redundant_type_annotations` * chore(trie): Replace redundant clone of entries in `InitializationJob` (op-rs/op-reth#644) Closes https://github.com/op-rs/op-reth/issues/631 - Puts off assigning variables until right when needed - Optimizes away redundant clone of entries on batch insertion - Updates symbol name to better show initialising is a batching algorithm * chore: moved proof initialization to `reth-optimism-node` (op-rs/op-reth#640) Closes op-rs/op-reth#612 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * chore: separate initial state store trait (op-rs/op-reth#647) Closes op-rs/op-reth#643 * chore(test): support reth as in sequencer and validator (op-rs/op-reth#650) This PR added support to run op-reth in sequencer and op-reth(with proof) in validator for the e2e test. Note: In this setup pruning test isn't supported(at this moment) since this test is falling outside the proof window of reth. --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: fix flaky e2e test (op-rs/op-reth#653) Closes op-rs/op-reth#528 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore(test): dep dump (op-rs/op-reth#654) Co-authored-by: Sadiqur Rahman <sadiqurr8@gmail.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> Co-authored-by: Himess <95512809+Himess@users.noreply.github.com> Co-authored-by: Himess <semihcvlk53@gmail.com> Co-authored-by: jagroot <4516586+itschaindev@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Julian Meyer <julianmeyer2000@gmail.com> * chore(op-reth/historical-proofs): fix compilation errors * kona: Remove IndexedBlobHash usage and simplify blob handling (#19081) * Refactor blob handling to remove indexed blob references * Refactor Blob Hint Tests to Improve Clarity and Readability * Add doc comment formatting fix to blob provider * Fix rustdoc syntax for trait documentation * refactor(dsl): introduce InitMessage and ExecMessage structs (#19227) * refactor: add InitMessage struct and update SendInitMessage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: update SendRandomInitMessage to return InitMessage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: update invalid_message_reorg_test.go to use InitMessage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: update cross_message_test.go to use InitMessage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: update interop_happy_tx_test.go to use InitMessage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: update interop_mon_test.go to use InitMessage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: update upgrade/post_test.go to use InitMessage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: update remaining op-acceptance-tests to use InitMessage Updates upgrade/pre_test.go and interop_msg_test.go to use the new InitMessage pattern. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: update kona supervisor tests to use InitMessage Updates all kona supervisor test files to use the new InitMessage pattern: - message/interop_happy_tx_test.go - message/interop_msg_test.go - pre_interop/post_test.go - pre_interop/pre_test.go - rpc/rpc_test.go Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: add convenience accessors to InitMessage Add BlockNumber(), TxHash(), and BlockHash() methods to InitMessage for direct access to commonly-used Receipt fields. Update all test files to use these new accessors instead of accessing Receipt fields directly. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: simplify SendExecMessage to accept InitMessage Update SendExecMessage and SendInvalidExecMessage to accept InitMessage directly instead of IntentTx. Remove the eventIdx parameter which was always 0. This simplifies the API and makes the relationship between init and exec messages clearer. Updated all call sites across op-acceptance-tests and kona tests. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: introduce ExecMessage struct Create ExecMessage struct to encapsulate exec transaction intent and receipt, mirroring the InitMessage pattern. Add convenience accessors BlockNumber(), TxHash(), and BlockHash(). Update SendExecMessage and SendInvalidExecMessage to return ExecMessage. Update all call sites across op-acceptance-tests and kona tests. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Introduce BlockID method. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Simplify logging of init and exec messages. --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> * op-acceptance: Add tests to prove validation of valid interop messages. (#19229) * op-acceptance: Add tests to prove validation of valid interop messages. * kona-proofs: Fix interop header_by_number lookup. Previously it would often enter an infinite loop because it would always look up the current block header and return that, never making any progress. * chore: update to Reth 1.11.0 commit (#19240) * chore: update to Reth 1.11.0 commit * Use tag instead of rev for reth deps * chore(rust/op-reth): op-reth v1.11.0 (#19247) * feat(flashblocks): implement speculative flashblock building (#18995) * chore: migrate docker images to oplabs GCP registry and fix prestate artifact output paths (#19251) Move kona-node, op-reth, and related docker image references from ghcr.io (op-rs/kona, paradigmxyz) to the oplabs-tools-artifacts GCP registry. Also fix the prestate build output directory to use an absolute path and update CI to write artifacts to a dedicated per-kind directory. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(flashblocks): add transaction caching to avoid re-executing unchanged transactions (#19030) * ci: disable incremental compilation and bump rust cache version (#19278) * ci: disable incremental compilation and bump rust cache version - Set CARGO_INCREMENTAL=0 in rust-setup-env to disable incremental compilation for all Rust CI jobs, reducing cache size and improving reproducibility - Bump rust build cache version from 15 to 16 to invalidate stale caches - Use a YAML anchor in main.yml so the cache version only needs to be set once Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(ci): remove stale ci jobs * ci: pin nightly toolchain and add weekly bump job - Add c-rust-nightly-version pipeline parameter (pinned to nightly-2025-11-01) to prevent surprise breakages from transitive deps incompatible with the latest nightly (e.g. shellexpand-3.1.1) - Update rust-install-toolchain to link a pinned nightly as "nightly" so existing `cargo +nightly` commands keep working - Replace all hardcoded `toolchain_version: nightly` with the parameter - Add rust-bump-nightly-pin job that opens a PR each week bumping the pin to the latest available nightly - Add scheduled-rust-nightly-bump workflow triggering on build_weekly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * kona/protocol/derive: handle "blob not found" correctly (#19328) * kona/protocol/derive: handle "blob not found" correctly * lint * lint * add a block number or hash in the error message * add named fields to BlobNotFound err * just fmt-fix * clippify * Simplify using inspect_err * simplifications * fix(kona-node): Map BlockNotFound errors to ResetError for reorg recovery (#19344) * kona/protocol/derive: handle "blob not found" correctly * lint * lint * add a block number or hash in the error message * add named fields to BlobNotFound err * just fmt-fix * clippify * Simplify using inspect_err * simplifications * Map BlockNotFound errors to ResetError for reorg recovery When an L1 or L2 block disappears (typically due to a reorg), retrying will never succeed. Convert these to ResetError so the pipeline can recover instead of stalling indefinitely. * just fmt-fix * Distinguish hash vs number lookups in BlockNotFound handling Hash-based lookups indicate a reorg (block removed) and require a pipeline reset. Number-based lookups indicate the block hasn't been produced yet and should be retried as a temporary error. * lint * ci: bump cimg/base from 2024.01 to 2026.03 The Docker daemon on CircleCI remote Docker hosts now requires API v1.44+, but cimg/base:2024.01 ships with Docker client v1.43. Bump to cimg/base:2026.03 to fix Docker API version mismatch errors in analyze-op-program-client and check-kontrol-build jobs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change ResetError::BlockNotFound to use BlockId instead of String This provides stronger typing and avoids unnecessary string formatting when constructing the error variant. --------- Co-authored-by: Matt Solomon <matt@mattsolomon.dev> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(kona/derive): preserve error kind from chain provider in BlobSource (#19357) BlobSource::load_blobs wrapped all errors from chain_provider.block_info_and_transactions_by_hash with BlobProviderError::Backend(e.to_string()).into(), which forced every error to PipelineErrorKind::Temporary regardless of the underlying error kind. This caused a regression: AlloyChainProvider was fixed in ee4d492a87 to emit PipelineErrorKind::Reset for BlockNotFound errors (mapping to ResetError::BlockNotFound). However, the Backend wrapping in BlobSource bypassed that fix, downgrading Reset to Temporary. During an L1 reorg when a blob-bearing block hash disappears, kona would spin retrying instead of triggering a pipeline reset, causing a liveness stall. The Go reference (blob_data_source.go:82-85) explicitly branches on errors.Is(err, ethereum.NotFound) and wraps it as NewResetError. Fix: use .map_err(Into::into) to preserve the error classification from the underlying ChainProvider::Error implementation. This mirrors the pattern already used correctly by CalldataSource. Fixes https://github.com/ethereum-optimism/optimism/issues/19354 * op-reth: bump reth/op-reth to v1.11.1 and MSRV to 1.92 (#19292) * ci: add cannon-builder image to Docker CI builds Add the kona cannon-builder (Rust MIPS64r1 toolchain) image to the branch and tag Docker build workflows. This publishes the image to the shared artifact registry so it can be consumed by prestate builds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * rust: bump op-reth to v1.11.1 and update cannon build infrastructure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: geoknee <georgeknee@googlemail.com> * refactor(op-devstack): remove backward compatibility type aliases to align on single ComponentID type (#18877) * stack: remove backward compatibility type aliases for Phase 6 cleanup Complete the ID type system refactor by removing all backward compatibility code. All code now uses ComponentID directly with typed constructor functions. Changes: - Remove 19 type aliases from component_id.go (L1ELNodeID, L2BatcherID, etc.) - Remove Kind = ComponentKind alias - Update KindProvider interface to use ComponentKind return type - Update context.go functions to use ComponentKind instead of Kind - Fix test files using incorrect constructor patterns The type system is now fully unified: - Single ComponentID type with kind, shape, key, and chainID fields - Typed constructors (NewL2BatcherID, NewL2ELNodeID, etc.) return ComponentID - ComponentKind enum for type discrimination - Simplified Matcher[E] interface with single type parameter * fix(op-devstack): drop misplaced kona tests and finalize componentid cleanup * fix(op-devstack): restore Go build after ComponentID refactor * chore(kona): remove deprecated supervisor crates (#19290) The OP Stack supervisor is deprecated. Remove all supervisor-related Rust crates, the supervisor binary, Go e2e tests, and Grafana dashboard recipe. Update workspace configuration, justfiles, and documentation. ci: remove kona-supervisor CI jobs and env vars Remove the kona-supervisor-e2e-tests job, kona-supervisor-e2e workflow, and RUST_BINARY_PATH_KONA_SUPERVISOR env exports from CircleCI configs. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(kona): add Karst hardfork support (#19372) * feat(kona): add Karst hardfork support Add Karst between Jovian and Interop in the fork ordering. Karst has an empty NUT bundle and requires empty activation blocks, matching the Go implementation in op-node (PR #19250). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(kona): chain Karst to Interop in op_fork_activation When karst_time is None, op_fork_activation should chain to Interop rather than returning Never. This maintains consistency between the is_karst_active() method and op_fork_activation(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(kona): don't chain Karst to Interop in op_fork_activation Revert ebd614d. Interop is an independent fork that doesn't participate in the sequential activation chain. Karst is the terminal fork before Interop, matching how Jovian was terminal before Karst was added. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix rust formatting * Chain karst activation to interop * fix(kona): address Karst review feedback * fix(kona): add missing karst_time to registry test configs and fix doc link --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * kona(derive): remove dead EthereumDataSourceVariant file (#19341) `variant.rs` defined `EthereumDataSourceVariant` and implemented `AsyncIterator` for it, but `AsyncIterator` is not defined anywhere in the kona codebase and the file was never included in `sources/mod.rs`. The type was therefore completely unreachable and unusable at runtime. `EthereumDataSource` in `ethereum.rs` already covers both the calldata and blob cases through the `DataAvailabilityProvider` trait, making `EthereumDataSourceVariant` redundant. Fixes https://github.com/ethereum-optimism/optimism/issues/19340 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(kona/derive): accept EIP-4844 type-3 batcher txs in CalldataSource (#19355) The `load_calldata` match on `TxEnvelope` used a catch-all `_ => return None` that silently dropped EIP-4844 (type 3) transactions. The derivation spec (derivation.md:504) and op-node's `isValidBatchTx` both explicitly accept type-3 transactions, so kona must do the same to avoid divergent L2 state in the pre-Ecotone window. Add a `TxEnvelope::Eip4844` match arm that extracts `to` and `input` like the other accepted transaction types, and update the corresponding test from asserting the tx is ignored to asserting its calldata is included. Closes #19352 Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(kona/protocol): add bounds checks in span batch decode to prevent panics on truncated input (#19361) * fix(kona/protocol): add bounds checks in span batch decode to prevent panics on truncated input Multiple span batch decode functions panic on truncated input instead of returning an error. This adds explicit length checks before each unsafe slice operation: - prefix.rs: decode_parent_check and decode_l1_origin_check now check r.len() >= 20 before split_at(20) - transactions.rs: decode_tx_sigs now checks r.len() >= 64 before indexing r[..32] and r[32..64] - transactions.rs: decode_tx_tos now checks r.len() >= 20 before indexing r[..20] On short input, each function returns SpanBatchError::Decoding(...) instead of panicking, allowing the batch to be dropped gracefully — consistent with Go op-node's io.ReadFull behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: fix rustfmt formatting in span batch decode Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: fix clippy field_reassign_with_default in span batch tests Use struct initialization syntax instead of Default::default() followed by field reassignment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: fix rustfmt formatting for struct initialization Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(kona/derive): copy origin field during Holocene BatchQueue→BatchValidator transition (#19360) * fix(kona/derive): copy origin field during Holocene BatchQueue→BatchValidator transition (BQ-12) On Holocene activation BatchProvider::attempt_update() transitions from BatchQueue to BatchValidator. It was copying l1_blocks but not the origin field, leaving BatchValidator.origin as None. Go's BatchMux.TransformHolocene() copies both fields (batch_mux.go:67-68): bs.l1Blocks = slices.Clone(bp.l1Blocks) bs.origin = bp.origin Without the origin copy, the first update_origins() call after transition always enters the self.origin != self.prev.origin() branch (None != Some(…)). This causes two failure modes: 1. Normal case: current L1 block is pushed onto l1_blocks again, creating a duplicate entry that corrupts the two-slot epoch window. 2. Lagging case: l1_blocks.clear() discards all transferred history, then next_batch() returns MissingOrigin.crit(), halting derivation. Fix: copy batch_queue.origin into bv.origin alongside l1_blocks, matching the Go reference implementation. Closes https://github.com/ethereum-optimism/optimism/issues/19356 * fix: apply rustfmt formatting fix Wrap long comment line in batch_provider.rs to satisfy nightly rustfmt check. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: rename test to reflect expected behavior The test verifies that origin IS transferred during Holocene transition, so rename from _not_transferred to _transferred. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove dangling reference to random issue ID. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(kona/derive): return Reset instead of Critical on blob under-fill in BlobData::fill (#19362) * fix(kona/derive): return Reset instead of Critical on blob under-fill in BlobData::fill When the blob provider returns fewer blobs than there are blob-hash placeholders (under-fill), BlobData::fill previously returned BlobDecodingError::InvalidLength which mapped to PipelineErrorKind::Critical — permanently terminating the pipeline and requiring operator restart. This is wrong. Under-fill is a transient condition: - The provider may not have all blobs cached/synced yet for a newly confirmed block - The L1 block may have been reorged out, causing the provider to return no blobs In both cases the correct response is to reset the pipeline and retry from the last safe L1 block, exactly as op-node does: fillBlobPointers wraps the same error as NewResetError at blob_data_source.go:110. Changes: - Add ResetError::BlobsUnderFill(usize, usize) to pipeline.rs - Add BlobProviderError::NotEnoughBlobs(usize, usize) to sources.rs, mapped to ResetError::BlobsUnderFill.reset() in From<BlobProviderError> for PipelineErrorKind - Change BlobData::fill return type from Result<bool, BlobDecodingError> to Result<bool, BlobProviderError>, returning NotEnoughBlobs on under-fill - Update blobs.rs caller to use the simplified .into() conversion - Update tests to assert under-fill produces a Reset-level error Fixes https://github.com/ethereum-optimism/optimism/issues/19359 * fix(kona/derive): remove unused BlobProviderError import The refactor to use Into::<PipelineErrorKind>::into made the direct BlobProviderError import unnecessary, triggering a clippy warning. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: apply cargo fmt to blobs.rs import ordering Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(kona/derive): address review feedback on blob under-fill PR - Wrap BlobProviderError in ResetError::BlobsUnderFill instead of duplicating fields - Use named fields for BlobProviderError::NotEnoughBlobs - Remove unnecessary map_err in BlobSource fill loop - Add Clone to BlobDecodingError and BlobProviderError Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * chore(rust/op-reth): bump op-reth to 1.11.2 (#19472) * fix(kona/derive): make syscfg update failure non-fatal in L1Traversal (#19358) * fix(kona/derive): make syscfg update failure non-fatal in L1Traversal When `update_with_receipts` fails to apply system config changes from L1 receipts, both `PollingTraversal::advance_origin` and `IndexedTraversal::provide_next_block` were returning a Critical pipeline error, permanently halting the pipeline. op-node's reference implementation (`l1_traversal.go:78-82`) treats this failure as non-fatal with the comment: "failure to apply is just informational, so we just log the error and continue." This commit aligns kona's behaviour with op-node: on syscfg update error, log a warning and continue advancing the origin rather than halting. Fixes https://github.com/ethereum-optimism/optimism/issues/19353 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: apply rustfmt to indexed.rs test line added in this fix The block2 struct literal added in this fix exceeds the 100-char line width limit. Wrap it to comply with rustfmt requirements. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(op-devstack): simplify acceptance testing around direct preset constructors (#19452) * refactor(op-devstack): simplify acceptance testing around direct preset constructors fix acceptance and kona e2e test harness regressions fix same-timestamp cycle test and kona shared runtime wrappers address review feedback and restore devstack test coverage fix(ci): stabilize rust e2e websocket tests * refactor(op-devstack): validate preset option support * fix(ci): reduce rust subprocess log volume * fix(kona/derive): add over-fill check in BlobSource::load_blobs (#19364) * fix(kona/derive): add over-fill check in BlobSource::load_blobs After the blob-pointer fill loop completes, add a post-loop check: if `blob_index < blobs.len()` the provider returned more blobs than were requested. Return `ResetError::BlobsOverFill` (→ PipelineErrorKind::Reset) rather than silently discarding the extras. This mirrors op-node's `fillBlobPointers` check at blob_data_source.go:162-163 which returns `fmt.Errorf("got too many blobs")` wrapped as `NewResetError`. Over-fill can occur with buggy blob providers (e.g. third-party RPC services that ignore the requested hash list) or in rare L1 reorg scenarios where the blob set shifts between hash collection and fetch. Changes: - Add `ResetError::BlobsOverFill(usize, usize)` variant to `pipeline.rs` (symmetric with the existing `BlobsUnderFill` variant). - Import `ResetError` in `blobs.rs` and add the post-loop guard. - Add `should_return_extra_blob` flag to `TestBlobProvider` for testing. - Add `test_load_blobs_overfill_triggers_reset` regression test. Fixes https://github.com/ethereum-optimism/optimism/issues/19363 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: use function names instead of line numbers in Go references Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(kona/derive): use named fields for BlobsOverFill variant Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(kona/derive): rename blob_index to filled_blobs in BlobSource (#19480) Rename the `blob_index` variable in `load_blobs` to `filled_blobs` for clarity, as the variable tracks the number of blob placeholders that were filled rather than serving as a traditional index. Addresses review feedback from optimism#19364. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove references to Go implementation in comments Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(kona/derive): remove useless BlobProviderError conversion in map_err The `From<BlobProviderError> for PipelineErrorKind` impl already exists, so `?` handles the conversion automatically. The explicit `map_err` was redundant. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: add fix-rust-fmt Claude Code skill (#19488) * chore: add fix-rust-fmt Claude Code skill Adds a skill that fixes Rust formatting CI failures by running `just fmt-fix` with the correct nightly toolchain via mise. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply suggestion from @ajsutton * Update .claude/skills/fix-rust-fmt/SKILL.md Co-authored-by: Sebastian Stammler <seb@oplabs.co> * docs: link /fix-rust-fmt skill from kona CLAUDE.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: reference fix-rust-fmt skill in rust-dev.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Sebastian Stammler <seb@oplabs.co> * chore(rust/op-reth): bump reth to v1.11.3 (#19498) Bump all reth dependencies from v1.11.2 to v1.11.3 in the rust workspace, including op-reth crate versions and the Cargo.lock. * chore(cannon): migrate Makefile to justfile (#19474) * chore(cannon): migrate Makefile to justfile Migrate cannon build targets from Make to Just. The Makefile now delegates to just with a deprecation warning, preserving backwards compatibility for existing make invocations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cannon): add missing lint target and include justfiles in Docker context - Add `lint` to DEPRECATED_TARGETS and justfile (CI compatibility stub) - Copy justfiles/ into kona cannon-repro.dockerfile for deprecated.mk shim - Install `just` binary in cannon Docker build for the Make shim Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cannon): remove [default] attribute for just <1.38 compat The Alpine 3.21 just package is v1.37.0 which doesn't support the [default] attribute. Move cannon recipe to first position instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cannon): include justfiles/ in op-program Docker build context The cannon Makefile deprecated shim requires justfiles/deprecated.mk, which is resolved relative to cannon/ inside the Docker container. The op-program Dockerfile.repro.dockerignore was excluding justfiles/ from the build context, causing the cannon make shim to fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(cannon): update README to use just instead of make Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cannon): call just directly in cannon-repro.dockerfile Instead of going through the deprecated Make shim, invoke just cannon directly in the Docker build. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cannon): update Dockerfile.diff to use just diff-cannon directly The diff-%-cannon Make pattern target was converted to a parameterized just recipe (just diff-cannon VM). Update the Dockerfile to call just directly instead of through make, which would fail since the deprecated shim doesn't support pattern targets. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cannon): add diff-%-cannon pattern target to deprecated Makefile Preserves backwards compatibility for make diff-<vm>-cannon invocations (used by Dockerfile.diff and potentially other scripts) by translating the pattern to just diff-cannon <vm>. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: install just via system package manager in Dockerfiles - cannon/Dockerfile.diff: use `apk add just` instead of curl install script, drop unnecessary `make` dependency - cannon-repro.dockerfile: switch cannon-build stage from ubuntu:22.04 to golang:1.23.8-alpine3.21, matching the monorepo's Go builder image, so just can be installed via `apk add` instead of curl install script Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cannon): bump Go to 1.24.10 in cannon-repro.dockerfile The golang Docker image sets GOTOOLCHAIN=local which prevents automatic toolchain downloading. Since go.mod requires go 1.24.0, the 1.23.8 image fails to build. Match Dockerfile.diff which already uses 1.24.10. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove devnet-sdk and kurtosis-devnet (#19506) * chore(rust/op-reth): port historical proofs binary change (#19252) * feat: add binary entry point for external proofs in OP (op-rs/op-reth#222) Closes op-rs/op-reth#164 --------- Co-authored-by: Arun Dhyani <dhyaniarun7@gmail.com> * feat: add support for `eth_getProof` (op-rs/op-reth#257) Adds support for `eth_getProof` RPC method. This required reworking the launch command to still work with an alternative DB provider. Closes op-rs/op-reth#173 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * perf(trie): feature gate `reth-optimism-trie` metrics (op-rs/op-reth#282) Closes https://github.com/op-rs/bin/issues/281 - Feature gates `reth-optimism-trie` metrics - Moves cursor impls out of proofs module into new module `cursor` - Moves cursor factory impls into new module `cursor_factory` - Updates cursor factory impls to return cursor types with metrics wrapper if metrics feature is enabled * fix(test): Enable live collector tests with metrics feature (op-rs/op-reth#291) Closes https://github.com/op-rs/bin/issues/283 Enable live collector tests when metrics feature is enabled * feat: implement `debug_executePayload` (op-rs/op-reth#276) Closes https://github.com/op-rs/bin/issues/189 --------- Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * feat: live collector integration (op-rs/op-reth#306) Closes op-rs/op-reth#296 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Emilia Hane <elsaemiliaevahane@gmail.com> * fix: rebase conflicts (op-rs/op-reth#367) Fix conflicts rebasing onto latest upstream main * feat: implemented `OpProofStorage` Database metrics (op-rs/op-reth#407) Closes op-rs/op-reth#224 Closes op-rs/op-reth#387 --------- Co-authored-by: …

meyer9 marked this pull request as ready for review October 28, 2025 15:52

meyer9 requested review from dhyaniarun1993 and itschaindev as code owners October 28, 2025 15:52

Copilot AI review requested due to automatic review settings October 28, 2025 15:52

meyer9 requested review from emhane and sadiq1971 as code owners October 28, 2025 15:52

Copilot AI reviewed Oct 28, 2025

View reviewed changes

emhane added K-observability Kind: observability A-db Area: database A-trie Area: Merkle Patricia Trie M-good-first-issue Meta: good issue for first time contributors W-historical-proofs Workstream: historical-proofs labels Oct 29, 2025

emhane suggested changes Oct 29, 2025

View reviewed changes

emhane approved these changes Oct 30, 2025

View reviewed changes

meyer9 enabled auto-merge October 30, 2025 15:02

meyer9 force-pushed the meyer9/246-live-sync-metrics branch 2 times, most recently from c556815 to c65bb87 Compare October 30, 2025 16:16

feat: integrate metrics into live sync process

bb94c5c

Signed-off-by: Julian Meyer <julian.meyer@coinbase.com>

meyer9 force-pushed the meyer9/246-live-sync-metrics branch from c65bb87 to bb94c5c Compare October 30, 2025 16:19

meyer9 disabled auto-merge October 30, 2025 16:19

meyer9 enabled auto-merge October 30, 2025 16:20

meyer9 added this pull request to the merge queue Oct 30, 2025

Merged via the queue into unstable with commit d5895df Oct 30, 2025
40 checks passed

meyer9 deleted the meyer9/246-live-sync-metrics branch October 30, 2025 17:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: integrate metrics into live sync process#330

feat: integrate metrics into live sync process#330
meyer9 merged 1 commit intounstablefrom
meyer9/246-live-sync-metrics

meyer9 commented Oct 28, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Oct 28, 2025

Uh oh!

emhane Oct 29, 2025

Uh oh!

meyer9 Oct 29, 2025

Uh oh!

emhane Oct 29, 2025

Uh oh!

meyer9 Oct 29, 2025

Uh oh!

emhane Oct 29, 2025

Uh oh!

meyer9 Oct 29, 2025

Uh oh!

emhane left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		#[derive(Debug, Clone, Default)]
		/// Counts from storing trie updates and post state.

Conversation

meyer9 commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

emhane Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

meyer9 Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

emhane Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

meyer9 Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

emhane Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

meyer9 Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

emhane left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

meyer9 commented Oct 28, 2025 •

edited

Loading